The wide-spread adoption of multimedia services, broadband internet, and cloud services, among others, have driven the demand of high transmission capacity of optical systems such as short-reach links, access, and metro networks. Typically, it requires optical transmission systems with high data rates (e.g., up to 1 Tb/s and beyond). To meet the ever increasing demands, access and metro networks are moving from the classic spectral inefficient single channel non-return to zero (NRZ) modulation to spectral efficient advance modulation formats such as wavelength-division multiplexing (WDM). The spectral efficient modulation formats, however, provide many distortions in their communication channels when used in optical systems.